In recent years, in the fields of VR (virtual reality) and CG (computer graphics), technologies for deforming a face image has been used for the purpose of providing an expression. For example, by detecting a change in a facial expression of a certain person and then applying the same change to a face image of the same person or a different person, a personified agent is provided which changes an expression in synchronization with a particular person.
There are mainly two trends in such a facial expression synthesis technique. One of them involves a method of treating as a physical model the skin surface moving in conjunction with muscles of facial expression under the skin of the face and solving motion equations based on action of muscles and bones to thereby move a wire frame model corresponding to the skin surface. This kind of conventional technique is described, for example, in “Analysis and Synthesis of Face image Sequences Using Physical and Anatomical Models”, (IEEE Trans. PAMI, vol. 15, no. 6. pp. 569-579, June 1993) by D. Terzopoulos K. Waters. The other trend involves a method of subjecting a wire frame model directly to regular geometric deformation to thereby create an expression. This conventional method is described, for example, in Japanese Laid-Open Patent Application (JP-A-Heisei, 8-96162).
On the other hand, a FACS (Facial Action Coding system), as an expression statement method, proposed by a psychologist Ekman et el., is widely used for the study of expression in the fields including the information engineering (see for example, “Facial Action Coding System”, (Consulting Psychologist Press, 1977), by P. Elman and W. V. Friesen). The FACS is a method of, based on anatomical insight of facial muscles, comprehensively and objectively stating a facial action in a total of 44 action units (AU) including “Lift up the inner eyebrow”, “Draw both ends of the lip horizontally”, “Open mouth wide”, “Make narrow eyes” and the like; and it is said that all human expressions can be stated with AU combinations. The FACS is, in a typical example, used in combination with a method of subjecting a wire frame model directly to regular geometric deformation to thereby create an expression.
As described above, the facial expression synthesis techniques conventionally includes the method of adopting geometric deformation and the method adopting a physical model. The method of adopting geometric deformation has an advantage that it is capable of performing high-speed processing with relatively small calculation amount since the rules for expressing an expression are only defined by point movement, but suffers from difficulties in reflecting a personal feature on the face model and also inability to provide a natural expression. On the contrary, the method of adopting the physical model is capable of directly expressing the action of muscles of facial expression, and thus is capable of smoothly expressing an expression creation process and also is capable of supporting various face actions by changing the arrangement of muscles of facial expression. However, this method suffers from a problem that the calculation amount becomes huge.
Related to the above mentioned matters, a face image synthesizing device is disclosed in Japanese Laid-Open Patent Application (JP-A-Heisei, 3-74777). In this conventional technique, a movement parameter and an expression parameter based on a human instinctive or unconscious action are added to face image information including an expression parameter and a movement parameter. A face image is synthesized based on the face image information.
Also, device for reproducing and displaying expression code is disclosed in Japanese Laid-Open Patent Application (JP-A-Heisei, 6-76044). This conventional device has an expression occurrence condition input device. An arithmetic processing unit 2 provided with an expression code reproducing means 21 which calculates extents and directions of movement and form changes of mobile parts such as the forehead, eyebrows, and the mouth based on medical analysis of the expression in accordance with the indication signal from the expression occurrence condition input device and reproduces an expression element code. An expression display device which reproduces and displays the form of the face in accordance with the expression element code reproduced by the arithmetic processing unit and moves and deforms the forehead, eyebrows, eyes, the mouth, etc.
Also, an expression encoding device is disclosed in Japanese Laid-Open Patent Application (JP-A-Heisei, 6-76058). In this conventional expression encoding device, a picture input device obtains face picture as a electric signal, and generates a certain face picture data by encoding the face picture. A feature location extraction processing unit receives the face picture data and extracts a feature location picture based on a feature location extraction condition. An expression element extraction processing unit extracts expression elements from the feature location picture based on a related rule between prescribed expression elements and the feature location. An expression element quantization processing unit calculates an expression element code from the expression element information based on a prescribed expression element quantizing rule by performing the expression element quantization processing. A storage unit stores the expression element code. An expression element code output device outputs the expression element code held in the processing device.
Also, a face image generating device is disclosed in Japanese Laid-Open Patent Application (JP-A-Heisei, 8-305878). In this conventional face image generating device, a face image generating unit generates a face image by selecting the parts image forming the respective parts of the face. A face image providing unit provides the change of expression to the face image by executing the image deformation, the movement of displaying position and display/non-display processing for the parts forming the face image in response to the appointed expression.
Also, a device for generating expression animation is disclosed in Japanese Laid-Open Patent Application (JP-P2002-304638A). In this conventional expression animation generating device, an expression difference data extraction part extracts feature points from an expression face sample of an object person, divides the expression face sample into parts according to the feature points, and finds differences of the shape and texture accompanying expression transition variation between two specified different expressions to obtain difference shape data and difference texture. An operation pattern setting part specifies an operation pattern function for prescribing time variation by the parts. A time-series expression image composing process part generates time-series expression shape data and time-series expression textures by specific time intervals according to the operation pattern function, difference shape data, and difference textures. A expression animation generating unit generates a expression animation by continuously displaying the time-series expression image by specific time interval, which is generated by mapping the time-series expression texture to the shape based on the time-series expression shape data.
Also, device for expression composition is disclosed in Japanese Laid-Open Patent Application (JP-P2002-329214A). In this conventional device for expression composition, the computer graphic technique is applied. This composition method includes the steps of: inputting a face expression model as face feature point data; extracting a nearly elliptic sphincter area surrounded with the sphincter of an expression model and prescribing the area above the major axis of the sphincter area as the semi-elliptic area surrounded with the major axis and a semi-elliptic shape; calculating the moving vector at a feature point included in the semi-elliptic area when the expression of a face change; and composing a expression model after expression change by moving the feature point based on the moving vector.